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ABSTRACT 



The purpose of the Web Retargetable Course Generation 
System, or "Recourse, " developed at Worcester Polytechnic Institute 
(Massachusetts) is to facilitate both distance and on-campus learning via the 
World Wide Web. Recourse's features include the ability to retarget a Web 
course for different levels of students, multi-user chat rooms, a secure 
grading system, bookkeeping tools, a map generator that allows students a 
birds-eye view of where they are in the course pages, a quiz feedback system, 
and a course bulletin board (news group) . This paper discusses the experience 
of teaching two online Web courses using these tools and describes the 
assessment procedures for evaluating the courses. Results are summarized 
related to computer knowledge background, student behavior, student attitude, 
student satisfaction, and course material. It is concluded that, while class 
satisfaction remained high, the formal assessment procedures indicated 
possible areas of improvement. (AEF) 
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Abstract: Virtual teaching via the Web is becoming commonplace. Tools to better enable this activity are beginning to 
appear. However, little formal assessment has been done to determine effectiveness of such tools nor the effectiveness of 
such distance learning. In this paper, we describe experiences teaching online Web courses and a set of formal assessment 
procedures for evaluating such courses. The courses, the tools and the assessment procedures have evolved over multiple 
teachings of the same two courses over the last three years both in the US and in Europe. One course is a Web Publishing 
course for non-computer science majors. The other is a Web Programming course for computer science majors. Statistics 
for both graduate students and undergraduates are included. 



1. Introduction 

Teaching courses via Web materials has new teaching issues plus old issues in a new setting: Just as in 
traditional courses, TA's and other assistants are needed. Traditional tasks (officer hours) as well as non- 
traditional tasks (staffing Chat Rooms) are needed. They are needed for maintenance of class pages, answering 
student questions - asynchronously via email and synchronously by holding ’’office hours” in Chat Rooms. 

While routine homeworks can be graded, recorded and responded to automatically, good software tools to 
enable this are just being developed. We have just developed and tested such tools. In the versions of the 
courses assessed here, all homeworks were graded by hand electronically and results emailed to the students. 
As will be described, this does not work well. 

When instructors teach a course for the second (third, fourth, ...) time, they reorganize existing material to 
make it appropriate for the current class. In traditional mode, this may include adding and deleting material, 
creating new projects, quizzes and assignments, refocusing for a different audience, etc. We have developed 
software to facilitate these tasks, but have yet to test them. Thus, all changes to these courses from previous 
versions was done by hand, checking and editing the course pages. 

The Web provides poor facilities for searching and navigating. Supplemental tools were developed and used in 
summer 1997 for the first time. 

We group our tools into a system called ReCourse [Lemone, 1996]. It also has been evolving over the last 
three years. It is a Web Retargetable Course Generation System whose purpose is to facilitate both distance 
and on-campus learning via the World Wide Web. By ’’retargetable", we mean the process of changing the 
Web course to "target" it for a different term or audience. 

Recourse’s features include: 

• Ability to retarget a Web course for different levels of students. A user-friendly editor allows instructors to 
add appropriate tags to HTML documents. Students then see only the parts of the pages appropriate for 
their level. 
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• Multiuser chat rooms to facilitate synchronous student, instructor, and TA communication. 

• A secure grading system allowing instructors to record grades and students to view their own grades. 

• Bookkeeping Tools such as a Hypertext Link Check to ensure that all internal and external hypertext 
references are valid, Search facilities, and Content Update tools to allow global updating of course pages 
(e.g., changing the term and date headers, course icons etc.) 

• A Map Generator to create a semi-static site map of the pages to allow students a birds-eye view of where 
they are in the course pages. This tool is run periodically by either the instructor or TA's when changes 
have been made in the organization of the course pages. 

• A Quiz Feedback system. 

• A course bulletin board (news group). 

The system can be entered as an administrator who installs the tools, as an instructor who sets up things such 
as the grading pages, generates the site map, checks for dead links etc., or as a student who can access the 
news group, the site map, his/her grades etc. 

This paper reports on the results of teaching using these tools, rather than on the tools themselves. More 
information on the tools can be found at http:/. www.webrecourse.com. 

2. Instructional Model 

People have been teaching courses via the Web for a number of years now. Sometimes the Web is used as a 
supplement to the class. Sometimes it is where the class takes place. We have experimented with a number of 
models and instructional designs and have learned and are still learning about the impact on student learning 
and faculty productivity of these models. In this paper, we describe results of teaching two summer courses 
almost entirely online. There was one meeting at the beginning where students met each other and the 
instructor, and the course format was discussed. At a final meeting at the end of the course, students presented 
the projects they had created during the course. 

A pretest was administered at the first class and a posttest with the same questions was administered at the 
final meeting. We describe these assessments and their results. 



2.1 Instructional Design 

Recourse is a Web-based system used in conjunction with Web course pages. It presumes course pages exist in 
a directory, and that there is a "root node" (home page); other pages are connected as links in the typical web- 
like architecture. Future enhancements can facilitate this creation. A typical course would have a number of 
modules representing the major topics in the course. Links also exist to the course information - email and 
phones of the instructor, TA and graders, Syllabus, Class list - with references to their home pages (if any) and 
their email addresses - Project deception (if any), and grading. 

The two courses, Network Publishing (http://cs.wpi.edu/~kal/netpub) , a Web Publishing course for non- 
computer science majors, and Electronic Documents (http://cs.wpi.edu/~kal/elecdoc), a Web Programming 
course for computer science majors were similar in format: a number of modules of information for which they 
sent in weekly homework, weekly labs which taught the publishing (page creation) and programming (Client 
and Server languages) aspects of the course, and a significant project which could be done singly or in groups. 
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2.2 Educational Technology 



Although the Web courses may be used within the classroom structure, they were developed for a distance 
learning model. Having taught this way for three summers, we have developed and incorporated techniques to 
facilitate distance learning: multiple (Web) references and weekly homeworks for reinforcement of the 
material, personalized responses when homework is submitted, and ’’presence" (asychronously via email, 
synchronously via Chat Rooms). In addition, the tools include automatic feedback on homework and birds-eye 
views of pages so that students can see where they are in the material and find other information more quickly. 



2.3. Comparison with Other Instructional Models 

Non Web-based distance learning models have relied on videotapes and broadcasts. While some Web courses 
have been taught synchronously via White Boards, etc., the technology just isn't sufficient yet. Our model is 
primarily asynchronous, allowing both the instructor and students to work at their own place, rate, and time. 
Our assessments included questions evaluating these features. 

Most Web-based courses are created and maintained by the instructors, perhaps with TA help. Few systems 
exist to aid the teaching of Web courses. WebCT [Goldberg 96, http://homebrew.cs.ubc. ca:8080/] comes the 
closest to ReCourse, but it lacks the "retargeting" facilities: when a course is retaught, it needs to be changed, 
updated, etc. Web courses take a phenomenal amount of time to develop, update and maintain. Tools to reuse 
material are needed. We know of no other system that addresses this retargeting issue. 

It was our hope that productivity would improve for the instructor and students due to: 

• TA help in chat rooms, a bulletin board and email. We spent hours each week, responding to email in the 
past. Sometimes, we could not respond in a timely manner. Support personnel are needed for distance 
learning in many of the same ways that they are needed for traditional classes. In fact, students may need 
more online personal contact from course personnel to overcome the lack of personal presence. The 
bulletin board was not ready for the summer, and perhaps because of this, the email quantity was again a 
major problem for the instructor and staff. 

• Automatic grading of weekly homeworks. We use routine assignments to encourage reading and 
assimilating of the course material. We grade them ourselves and send students feedback and their scores 
via email. Again, this takes a few hours/week. The automatic test system will ease this. We did not have 
this fully tested and integrated for security this summer, but it will be used this Fall. The conclusions will 
discuss the very real need for such a system as well as a potential drawback. 

• The Bookkeeping Tools allowed the instructor to quickly find dead links, and to generate a site map; 
students were able to use this site map to “see” where they were in relation to the rest of the pages. The search 

tool (suggested by a previous class) was extensively used. 

• The retargeting tools will enable the instructor to create the next version of the class in far less time than 
we presently spend. They were not used for the summer versions assessed here. 

• Instantaneous feedback to students on their homework. For this version, just a personalized 
acknowledgment page was sent; the next version of the course will send back a graded page with correct 
answers and a paragraph of explanation for each question. Issues of security (the answers were accessible 
via a Java program) prevented their use this summer. 

• Automatic and secure access to student grades (for students and the instructor.) Again, this was not fully 
secure for the summer, and students expressed a strong desire for it. 
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3. Assessment Plan 



We were funded by the Davis Educational Foundation to develop and perform statistically significant 
assessments on these classes. 



3.1 Procedures and Instruments to Measure Effectiveness 

We have been using student questionnaires for the last 3 years. There is a preliminary questionnaire, and a 
post questionnaire for each course. One term, students filled out weekly assessments. Interestingly, students 
have always filled out these electronic Web forms even when they ran a week or two behind. We've never 
gotten anywhere near this response with paper questionnaires! 

However, we decided more formal assessment procedures were needed. 

3.2 Description of Control Groups and Comparison Tools 

We assessed the effectiveness of the Web courses and the Recourse software in the summer versions of two 
classes: Electronic Documents and Network Publishing. The Network Publishing group are less technical, 
more writing and publishing-oriented (in theory). The Electronic Documents group are Computer Science or 
Computer Engineering majors (or those with strong computer backgrounds.) We compared these groups, not 
with each other, but with information gathered via a pretest and a posttest. We gathered and compared issues 
such as (1) background, (2) behavior, (3) attitude, (4) satisfaction, and (5) knowledge and skills gained. 



3.3 Pre/Post Analysis 

For the preliminary questionnaire, we asked questions about their background and interests, e.g., questions 
concerning Web experience. For behavior, we asked questions such as the number of hours per week they 
planned to spend. For attitude, we asked questions such as whether they (would/would have) prefer/red the 
course to be taught in the traditional manner (as opposed to online). For satisfaction, we asked questions such 
as helpfulness of the instructor and whether they think/thought the course to be useful. 

Finally, both the pretest and the posttest included 100 objective (mostly multiple choice) questions relating to 
the material. Because of the large number of questions, it was hoped they would not remember a significant 
number of questions when studying for the posttest. 

We also used the WPI standard course evaluation form (The first 14 questions indicate an overall measure of 
satisfaction, and the very last question indicates self-perception of learning.) These results are not yet 
available. 



4. Outcomes 

We summarize the results of the various categories. 

4.1 Measurable Outcomes 

Background: Not surprisingly, the non-computer science majors showed less preliminary knowledge of Web 
related information: few had created Web pages although most had used the Web. About 3/4 of the computer 
science majors (Electronic Documents course) had a Web page, and about % indicated some knowledge of 
client and server programming languages (primarily Perl, JavaScript, and Java) 




5 



Behavior: On the pretest, most students indicated they planned to spend 15-20 hours/week with a few 
planning fewer hours. On the posttest, the majority indicated that they spent in excess of 20 hours a week with 
a few spending less. 

Similarly, most students expected to spend 3 days/week before the course, but indicated having spent 4 or 
more on the posttest. 

Students were split on the pretest as to whether they planned to print out the course pages or not; most 
indicated on the posttest that they did print out at least some of the pages. 

On the pretest, students were split between spending 5-10 hours and 10 or more hours “surfing” the web. 

The totals were actually down in the posttest. 

Most students didn’t know whether they would use the chat room or not before the course. Most of the 
more technical students in the Web Programming course said they did not use it, while many of the less 
technical Network Publishing students used it more - they also came to the “in person” office hours. Both the 
TA’s and the instructor used the chat room, and they all indicated they thought it was an effective way to deal 
with students. 

Attitude: Most students “liked the idea” of taking a course online as opposed to the traditional in class model 
as indicated on the pretest with a few circling “not sure.” On the posttest, everyone indicated they like it with 
1 student indicating he/she “wasn’t sure” he/she would take such courses in the future. Everyone else wanted 
to take more such courses. Students Indicated on both the pretest and the posttest that they did not believe the 
course could be done with no meetings at all. 

Satisfaction: Most, but not all students indicated that the course objectives were clear both before and after the 
course. Almost everyone felt the course was well organized. Most, although not all, students expected and 
found the material challenging and interesting. Not everyone felt the instructor was helpful, while most 
expected her to be so. Everyone expected to be able to apply the materials and skills learned to their 
professional lives. Most, although not all, felt the homeworks and the assessment (posttest) measured their 
knowledge of the material. Only one student felt he hadn’t learned a lot in the course. 

Course Material: No one knew many of the answers for the pretest. Posttests were, of course, much better 
although it will be interesting to compare these results with those of the next course (none of the tests are 
allowed to circulate,) 



5. Conclusions 

Class satisfaction has been high in the past, and continued to be so. Students seem to like taking a course 
(mostly) on their own in the summer. Whether this model would be successful during the year or for many of 
their courses remains speculation. Although not as objective as times and correct answers to a question, 
satisfaction can still be measured, at least qualitatively, and reported on. Comparison of the student's desired 
outcome ("What do you hope to learn in this course?") described on the pretest with the actual outcome ("Did 
you learn (less than/more than/ etc. ) what you hoped to learn") on the posttest, is an important measurable. 
(We email back right away when a desired goal is unrealistic for the course.) 

Nevertheless, the formal assessment procedures indicated possible areas of improvement. Given that the 
instructor was spending many, many hours/week on the course, it was disheartening to find out that some 
students felt they were not able to communicate well. A course bulletin board, better grading software and a 
better delegation of tasks among ta’s and instructors may improve this. 
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